Adaptive Probabilities in Stochastic Optimization Algorithms

Stochastic optimization methods have been extensively studied in recent years. In some classification scenarios such as text document categorization, unbiased methods such as uniform sampling have negative effects on the convergence rate, because of the effects of the potential outlier data points on the estimator. Consequently, it would take more iterations to converge to the optimal value for uniform sampling than non-uniform sampling, which normally allocates larger probability to samples with larger norm. Based on the intuition of non-uniform sampling, we investigate adaptive sampling in which we change the probability per iteration or per epoch during the execution of the algorithm. In this thesis, we start by introducing non-uniform sampling with Stochastic Gradient Descent (SGD), and then study the impact of different choices of stepsize on convergence rate as well as derive theorems for the convergence rate. We analyze Stochastic Dual Coordinate Ascent (SDCA) and obtain the upper bound on the number of iterations to reach a specific level of optimality. This is followed by investigation of adaptive strategies which change the probabilities according to some historical information. We provide theoretical analysis to prove that they further reduce the number of iterations needed to reach the same level of optimality. The adaptive variants of SGD and SDCA are discussed and compared to the existing counterparts. The results show that both AdaSGD and AdaSDCA outperform the existing models. We also provide two online variants which update the probabilities immediately to perform adaptive sampling without the need of additional passes through the data.